@GrigoryPervakov two points, let me know your thoughts.

1. For mutable image tags, the cached key (image string + pull policy) can go stale. Realistically for production use-cases I would assume it not be a problem, but if we want to handle it, will require some logic to invalidate the cache. An ideal cache key would have been image sha, but that requires the pod to be already running or a brittle talking to registry logic.
2. With respect to the original issue, caching should practically solve it. But the original scenario is still an edge case i.e. if job is complete and it fails to read a version. I considered auto re-creation of completed job, but I'm not sure if there are any other scenarios besides missing pod for failure. We could leave this as is for now i.e. manual remediation.

1. For mutable image tags, the cached key (image string + pull policy) can go stale. Realistically for production use-cases I would assume it not be a problem, but if we want to handle it, will require some logic to invalidate the cache. An ideal cache key would have been image sha, but that requires the pod to be already running or a brittle talking to registry logic.

Let's keep it as it works now. For a production environment, it makes sense to use a full version tag.
I don't see a good way to use image SHA without running a Pod, so I prefer to wait for any demand fixing it.

2. With respect to the original issue, caching should practically solve it. But the original scenario is still an edge case i.e. if job is complete and it fails to read a version. I considered auto re-creation of completed job, but I'm not sure if there are any other scenarios besides missing pod for failure. We could leave this as is for now i.e. manual remediation.

I thought about such scenarios, and I can't find any realistic cases where a successful Job produces invalid output, and this is not a persistent issue